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INTRODUCTION * 



Since emerging to prominence as a national issue in higher education some four years 
ago, the assessment of educational outcomes has been inextricably linked with 
accountability. Beginning as a reform movement internal to the academy with the 
publication of such reports as Involvement in Learning (NIE 1984) and Integrity in 
the College Curriculum (AAC 1985), the "high ground" of assessment was quickly 
seized by state policy makers who saw in assessment a powerful "lever for change" for 
improving quality in undergraduate education (ECS 1986, NGA 1986, Ewell 1985). 
Today, the majority of assessment activities occurring at American colleges and 
universities is due to state initiatives. Over half of the. institutions responding to a 
national survey on the issue in 1989 claimed that their primary motivation was an 
existing or anticipated state mandate (El-Khawas 1989). More decisively, in a fall 
1989 survey of state highe- education agencies, more than three-quarters of the states 
reported a student aissessment effort planned or in place (Ewell and Finney 1989). 

Many observers have seen in this pattern a change of agenda. Initial institution- 
centered efforts at assessment have been characterized as oriented toward "academic 
improvement," while external mandates reflect a demand for greater "accountability." 
Public policy makers, in contrast, have long contended that the notions "accountability" 
and "improvement" are inseparable, and that delivering quality will achieve 
accountability. At bottom, both positions are right. Both are concerned with quality 
improvement and see in assessment an essential tool in achieving it. Both are also 
convinced that "assessment" cannot do the job alone. Their difference lies in the fact 
that for many at the state level the concept of "accountability" itself has changed, and 
with it the appropriate roles that state-based assessment is expected to play. 
Increasingly, state leaders are willing to become more proactive about needed 
changes. And they are increasingly unwilling to leave the definition and delivery of 
quality to higher education institutions al. 

The nature of this change, and its implications for assessment policy, are the topics of 
this paper. Its "database" is a series of nine case studies of state-based assessment 
initiatives undertaken over the last two years by the Education Commission of the 
States (ECS). The first year of ECS case-study work on state assessment policy 
focused on five states implementing assessment (Colorado, Missouri, New Jersey, 
South Dakota and Virginia); here, appropriate analytical questions concerned the 
dynamics of policy formation and the pitfalls and politics of early implementation 
(Ewell and Boyer 1988). Case-study work in six states in 1988-89 (Florida, Illinois, 



•Though the conclusion of this paper are my own, I must gratefully acknowledge 
the superb contributions of my colleagues in the field: in 1987-88, Carol Boyer of ECS, 
and in 1988-89, Joni Finney of ECS and Charies Lenth of State Higher Education 
Executive Officers (SHEEO). Their insight, patience and persistence have made this 
project possible. 
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New Jersey, Tennessee, Virginia and Washington) examined a more basic set of 
issues. Among them were the emerging uses and impacts of assessment information 
at the state level, and the integration of assessment with other available state policy 
pechanisms. In each state, interviews were conducted with a range of constituencies 
including legislators, governors and their representatives, coordinating board members 
and staff, and institutional representatives. Results were summarized thematically, 
taking care to recognize that each state provides a unique political, fiscal, and 
historical context within which the substance of policy must unfold. What follows 
therefore is an attempt to chart some common ground at a high level of generality, 
but in a form that will hopefully be both authentic and informative. 



INGREDIENTS FOR A NEW ACCOUNTABILITY: 
A CHANGING VOICE 



The case for a "new accountability" is rooted in what appears to be a fundamental 
shift in the way policy makers perceive the public role of higher education. This shift 
is visible in a changing language-in particular, in the issues, symbols and overtones 
that policy makers call on when attempting to describe what they expect from higher 
education. Perhaps because it is most familiar, ihis rhetoric often begins with costs. 
Recent comments on higher education by a number of governors at the President's 
"Education Summit" in September 1989 (Chronicle of Higher Education 1989) 
centered on costs-a traditional concern-but demonstrated a growing impatience with 
traditional arguments about why increased investments were necessary. Such 
comments are echoed by a legislator in all nine ECS case-study states. "You've got to 
give something to get something," one summarized, "the era of saying *put the money 
on the stump and trust me' is over." 



On the face of it, such comments provide few grounds for a fundamentally altered 
conception of "accountability." Clearly, legislators and the public have always been 
concerned about the manner in which public funds are invested in higher education, 
and have occasionally erected elaborate and intrusive controls over resource allocation 
and expenditure. If legislative voices stopped with these concerns, there would be 
little new to report. But they are different in character in at least three ways. First, 
greater "accountability" is not for the most part a rhetoric of disinvestment. Indeed, it 
is seen as a critical key to continued budgetary expansion. Governors such as Kean 
in New Jersey and Gardner in Washington (among the most outspoken critics at the 
"Education Summit") and legislators such as Bragg iri Termessee, Peterson in Florida, 
and Doria in New Jersey (prime sponsors for major spending bills on higher 
education in their respective chambers) believe that further increments in investment 
cannot be sold to their colleagues without some kind of demonstrated performance in 
return. 

Secondly, concrete information about performance itself is increasingly advanced as 
the substance of "accountability" and as a key to achieving quality. Ironically,' this 
demand is seen as less intrusive by its authors than are detailed expenditure controls. 
Far from stifling institutional creativity, many legislators believe that a performance- 
based approach to achieving quality can be both flexible and enabling-consistent with 
emerging "best practice" in corporate management. Most are therefore genuinely 
puzzled when higher education balks at collecting and applying information about 
performance; indeed, many profess astonishment that such information is not already 
firmly established. 

Finally, if the substance of "accountability" is information about performance, its 
predominant form lies in a more proactive approach to public communication. For 
too long, many legislators claim, higher education has hidden behind residual 
deference and perceptions of complexity. In an atmosphere of rising costs and visible 
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shortfalls in what needs to be accomplished, this approach will no longer suffice. "I 
think that [higher education in this state] does not sufficiently understand the symbolic 
side of accountability," said one observer, "and they're missing a major opportunity," 

This changing voice can be succinctly summarized in terms of five policy imperatives 
that are both persistent and pervasive in the testimony of board members and elected 
officials across states: 

1- "Higher education is no longer an isolated enterp risp, " Unlike K-12 education, 
where deficiencies are seen as chronic and fundamental, higher education is 
perceived as unresponsive. Governors and legislators commonly made a 
connection between higher education and economic development or the 
demands of responsible citizenship.. Most in addition saw public colleges and 
universities as insensitive to individual students and their needs-particularly in 
undergraduate education. At the same time, they stressed that higher 
education had paid little attention to developing meaningful partnerships with 
K-12 education. Finally, many had been personally frustrated in their attempts 
to obtain concrete answers from higher education officials to what they 
considered to be relatively straightforward questions about performance. 

2- "You've got to give something to get something ." Most of those interviewed 
were prepared to expand investment in higher education, so long as it could 
demonstrate its commitment to quality and its responsiveness to public needs. 
For the most part supporters of higher education, they lacked the tools to build 
a credible case for budgetary expansion with their colleagues. Information 
about performance and quality improvement was seen as a necessary condition 
to leverage funds and not, as most institutional leaders feared, a disguised 
retrenchment device. 

^- "You're the professionals,,, vou tell ns." Behind a concern for information about 
performaiice, however, was little interest in dictating the details of 
accountability. The overwhelming sentiment among legislators was to allow 
higher education's !eadership-"those closest to the problem"--to define how and 
across what dimensions such information should be provided. The primary 
issue for higher education was seen less as one of specific performance than 
of overall credibility: most were willing to allow higher education considerable 
flexibility so long as it was accompanied by a visible commitment to greater 
responsiveness, 

^- "It can't be that complicatftd " An important perceived ingredient in 
credibility, however, was simplicity. Rather than masses of performance 
statistics, most legislators appear concerned with overall direction and a few 
carefully chosen "benchmarks" of perfo/mance. As a result, they are frustrated 
by the masses of uninterpreted statistics that form the growing substance of 
accountability. What they feel they need most is evidence that the enterprise is 
intelligently managed-that it kno'vs where it is going and how to get there. 
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Well-articulated goals and planning statements are as much a part of this effort 
as are a range of undifferentiated performance statistics. 

5. "YouVe about to miss the boat ." Finally, throughout all interviews there was a 
sense that available time in which to forge new policy is limited. Education is 
currently high on the public agenda, and it will not long remain so. By 
continuing to view accountability in terms of "business as usual" higher 
education is seen to be missing an important opportunity to make its case and 
to gamer new resources* Within another two or three years continuing 
unresponsiveness may force a shift in the political agenda to new tactics, and a 
shift in public attention to new issues. 

Behind these changing voices, I believe, is a fundamental shift in how higher 
education as a public investment is perceived and managed. Once viewed primarily 
as a provided "public service" whose chief concerns were equitable access and 
operational efficiency, public higher education is increasmgly being viewed as a 
strategic investment. As such, its major perceived payoff lies in such areas as 
enhanced economic development and increases in common skills among potential 
w6rkers and citizens. Wliile the traditional conception of accountability raises policy 
issues that are primarily distributional in character, the second embraces higher 
education as a productive system. And while the first is consequently focused largely 
on imperatives of maintenance and control, the second emphasizes the changes 
necessary to achieve a quality product. 

Concretely, this change is visible in at least three recent state policy trends that echo 
the shifts in rhetoric outlined above. The first is a trend toward directed investment, 
visible in a range of addition-to-base funding mechanisms whose intent is to channel 
resources in support of identified state priorities. Examples in the nine case-study 
states include the Governor's Challenge Grant program in New Jersey; "Centers [or 
Funds] for Excellence" grants in Colorado, Florida, Tennessee, and Virginia; and more 
targeted quality enhancement efforts such as the "Chairs of Excellence" program in 
Florida. Such marginal funding mechanisms are often tied to explicit evaluative 
procedures stressing improved instructional quality and return on investment. Indeed, 
many of them have been used to fund the development of broadly-conceived learning 
assessment and improvement efforts at selected pilot institutions. 

A second trend is a growing "trade-off* between greater management flexibility for 
higher education to address needed changes in return for information about results. 
This trend is visible in recent autonomy legislation granted to New Jersey's state 
colleges and in Hawaii's decision to remove its university 5yitem from the restrictions 
of operating under the state's rigid state financial and personnel controls in return for 
"performance assessment" administered through the legislative auditor's office. Not 
only does such a bargain reflect prevailing management wisdom with which legislators 
and state lay board members are increasingly familiar, but it also reflects the 
acknowledged failure of prior attempts to micro-manage change through direct 
intervention. 
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A final, highly visible, policy trend is an escalating demand for concrete information 
about student performance, visible in numerous emerging state-based assessment 
programs. All of the nine case-study states had or were implementing such programs- 
-rangmg from established testing and performance funding approaches in Florida and 
Tennessee to emerging, more institution-centered approaches in the remaining states. 
Despite their apparent diversity all nine state assessment approaches were initially 
conceived by policy makers as part of a range of integrated policy tools deployed in 
support of a coherent state strategy. All, at bottom, were concerned with inducing 
changes in instructional practice that resulted in improvements in quality. In more 
than a few cases, however, policy integration did not survive implementation. Faced 
with the intricacies of managing a complex, multi-faceted initiative, state authorities 
often treated assessment policy in isolation-a "train on its own track" (Ewell and 
Boyer 1988). And once on its own track, assessment proved at best an indifferent 
policy tool to induce institutional change. 

Repeated loss of coherence regarding the place of assessment in larger state change 
strategies demands renewed policy attention to the specific roles of state higher 
education governing and coordinating agencies. Above all, achieving accountability 
within a framework of quality improvement requires policy "brokering" between the 
external function of demonstrating return on investment on the one hand, and that of 
appropriately managing required institutional changes on the other. It is a proactive 
strategy, demanding both articulation and visible leadership. Viewed externally, 
"accountability" in this sense is primarily a question-6f communication, with a 
particular focus on providing evidence of responsible stewardship of public resources 
consistent with societal needs and priorities. Here the emphasis must increasingly 
shift from "accounting" for existing investments through expenditure control to 
leveraging new ones through demonstrated results; "assessment" becomes an effective 
way to sell the additional resources required to sustain a transformatory role for 
higher education. Viewed internally as a mechanism for directed change, the 
connection between "assessment" and "accountability" is also more dynamic. Rather 
than being used to police existing institutional instructional practice through inspecting 
Its products against minimum standards--a dominant current institutional metaphor for 
state-based assessment practice-assessment is increasingly viewed as a mechanism for 
actively shaping institutional agendas toward greater attention to instructional quality. 
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THE EXTERNAL AGENDA FOR ACCOUNTABILITY: 
MAICING THE CASE FOR HIGHER EDUCATION 



"Public accountability" has always involved a communications component. Its 
substance consists not only in having actually discharged assigned responsibilities but 
also in making an effective case that this has in fact occurred (Jones and Ewell 1987). 
Traditionally, however, such communication has been post hoc; that is, "accountability" 
reporting occurs after a set of activities takes place, primarily to ensure that they are 
consistent with original intent. 

The "new accountability" demands a much more proactive approach to communication. 
Firs^ it is founded as much upon articulating ends as upon justifying means. This 
implies using a range of mechanisms, including assessment, to present a clear picture 
of where the state's higher education system ini'^nds to go and how it intends to get 
there. Second, it involves providing broad penbrmance information keyed primarily 
toward the achievement of quality goals. This implies the ability to supply concrete 
bodies of evidence about student abilities and attainments, though it need not imply 
using common indicators across a diverse body of institutions. Finally, it involves 
recognition that increased resources can be obtained only if the first two conditions 
are met. As a result, the focus of communication is on the proactive use 
performance information to help higher education's political supporters make the case 
for increased funding. 

External requisites for accountability can be described in terms of three broad policy 
directives derived from the experiences of case-study states. In each case, the 
directive refers to an overall state strategy, involving partnership between higher 
education's political supporters and those (primarily in state agencies) responsible for 
overall policy direction. Though indirect participants, institutions must also be made 
aware of this strategy, and must recognize its potential collective benefits. 

1- Re-establishing Credibility. A first item on the agenda is to actively address 
diminishing levels of confidence in higher education felt by legislators, 
executives, and the wider public. Above all this requires clear recognition that 
while the perceived "problem" is generally one of quality improvement, it is not 
usually one of specific performance. Unlike K-12 education, higher education 
in the case-study states was not generally seen as an area characterized by 
extreme underperformance, requiring radical measures in response. "They don't 
think anybody's ripping them off," summarized one key executive staff member 
in Virginia. Most interviewees in contrast, noted that current perceptions of 
reasonable performance offered a unique opportunity for improvement. But 
many were quick to note that a primary issue was loss of confidence-what one 
legislative staffer aptly labeled "the symbolic side of accountability." For the 
most part, this was less a matter of demonstrating performance in detail than 
of building a credible case that higher education "knew what it was doing," was 
concerned about instructional quality, and was fundamentally well managed. A 



Washington legislator put it well: "We want to achieve a level of comfort on 
this issue so we can leave it alone." 

Equally clearly, legislative "discomfort" was founded upon a failure of 
traditional accountability mechanisms to build the required confidence. 
Budgetary control systems were often seen as inefficient and too much subject 
to manipulation: one legislator called them "a shell game." External approval 
mechanisms such as accreditation were seen as having "lost all credibility." 
Program review processes such as those conducted by Florida's Board of 
Regents or the Illinois Board of Higher Education (IBHE) were viewed more 
favorably, but were not directed clearly enough toward questions of 
instructional quality and return on investment. "When we mention program 
review [to legislators]," complained a higher education staffer in Illinois, "their 
eyes glaze over." 

What has been the roie of state assessment policy in helping to re-establish 
credibility? Most important was the simple fact that information about 
performance was being visibly collected with the promise that it be used. In 
New Jersey, Virginia and Tennesse<v-all states with initiatives that had been in 
place for at least three years-legislators knew very little about the way 
assessment was structured or what kinds of information was being collected. 
But they were sufficiently comfortable with the initiative itself that they also 
showed little interest in finding out. As a New Jersey legislator put it, "the 
fact of assessment is more important than what they end up doing." In all 
three states, unilateral action by boards and institutions to establish such 
programs helped purchase considerable discretion in how they should be 
structured. 

In states where legislative involvement in assessment was more direct, 
"credibility" payoffs depended upon a mixture of what was done and what was 
found. A major beneficiary of the "rising junior" CLAST examination in 
Florida, for example, was the state's community college sector. Here, 
satisfactory pass rates on the examination for AA graduates acted to counter 
charges of "grade inflation" ai J "watered-down classes" that had been frequently 
leveled at the community colleges. At the same time, state community college 
board staff were able to use evidence of deficiency to garner additional needed 
funds for developmental and college preparatory work. Not only did the scores 
themselves help make the point, but their ready availability aided legislative 
"confidence-building" by sustaining the impression that state board officials knew 
the nature of their problem and how to deal with it. This effect was echoed in 
Tennessee, where Tennessee Higher Education Commission (THEC) staff 
reported that information gathered through the state's "performance funding" 
program has had a major impact on their ability to talk credibly to the 
legislature. "I spend the first fifteen minutes [of legislative hearings] being able 
to talk about the progress that's been made," reported the Director of THEC, 
"and this is very impressive to them because you don't just lead with the 
budget." 
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Put simply, the visible presence of performance information "signals" good 
management. And with a few exceptions, it is good management, not specific 
performance that for most policy makers defines the substance of 
accountability. 

Expanding the Base of Resources . In the majority of case-study states, 
"assessment" first arose as an element of public policy in a context of resource 
expansion. Only in Missouri and South Dakota was the predominant tone of 
the initiative to achieve greater efficiencies on an existing base. Contrary to 
the fears and perceptions of most institutional observers, assessment was seen 
by the majority of state policy makers primarily as a means to politically make 
the case for increased revenue. "The public won't stand for any more tax 
increases without this," reported a key Tennessee legislator on the origins of 
that state's Legislative Goals process, ^you can't just raise dollars and spend 
them any more." 

Perhaps the most explicit linkage of this kind occurred in the state of 
Washington. Here, the Higher Education Coordinating Board's Master Plan 
included both a sophomore testing provision and a mechanism intended to re- 
establish on a new basis support for a notoriously underfunded higher 
education system. If adopted, the Master Plan's proposed funding mechanism 
would both expand the total pot available and would decrease controls over 
targeted expenditure. But such an approach eliminated "legislative self- 
assurance"; in the words of one legislator, Ve had to make sure that this 
funding wasn't just seen as an entitlement." Swapping funding for assessment 
was thus a necessary i>art of the strategy for selling the Plan politically. Given 
this context, lack of institutional enthusiasm was viewed with some puzzlement 
by state policy makers. It was clear to them that non-cooperation would not 
result in "punishment," but rather in "a major lost opportunity" for the 
institutions to improve their position. 

In part, the need for new methods to justify increased funding is due to the 
visible inadequacy of traditional arguments and traditional mechanisms for 
leveraging funds. In particular, legislators in several states exhibited impatience 
with traditional contentions (usually put in the mouths of Presidents at 
legislative hearings) that the primary need was for additional base funding "to 
uplift the entire system" of higher education. Not only did such arguments 
tend to substantiate the image of "entitlement" funding, but they also seemed to 
suggest that higher education leaders did not understand their own "production 
functions" sufficiently to know where and how to invest additional resources. 

At the same time, traditional mechanisms for justifying additional funding based 
on enrollment growth were no longer adequate. As one official put it in 
Virginia, a major current policy problem for higher education is that "we don't 
know how not to fund growth." In Tennessee, performance funding was first 
mooted because of fears about enrollments topping out and the consequent 
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inabili^' of the formula to raise additional dollars. The notion of performance 
fondini: vas less a mechanism for changing institutional behavior than a means 
of "raisixig legislative aspirations" that would be visible in increased overall 
funding. "We did have a naive notion that the institutions would do good 
things," recalled an original architect of the program, "but the shift to 
'assessment' was very much an afterthought" By demonstrating that it was 
possible to break the linkage beUveen dollars and enrollments and still be fair, 
noted another, performance funding "got us out of a mentality"; as a 
consequence, "it was a foot in the door for a lot of other things" to be 
undertaken in the realm of quality improvement Most importantly, in 
Tennessee performance funds are "not just ano±er eighteen million dollars." 
Instead, in the opinions of THECs leac rship, "they are the key to selling the 
other ninety-five percent." 

Given this linkage, it is no coincidence that some of the most visible 
assessment initiatives occurred in states that were consciously attempting to 
improve their competitive positions in higher education through a broad-based 
quality improvement effort In Florida, CLAST was only one part of a 
comprehensive package of reforms and investments initiated by Governor 
Graham in the mid-seventies, intended to move Florida from its position as 
"just another good Southern state." In New Jersey, the College Outcomes 
Evaluation Program (COEP) was also part of a range of Department of Higher 
Education (DHE) policy initiatives intended to move the state's public higher 
education system from a "backwater" to a position of national prominence. 

Acknowledging assessment's link to resource expansion, however, compels, equal 
recognition that in its absence assessment has little political constituency of its 
own. In Virginia, where the public purpose of assessment is to encourage long- 
term institutional improvement, additional authorization had to be written into 
the code for the State Council for Higher Education in Virginia (SCHEV); 
absent this authorization, it was felt that raising funds to support assessment 
would be difficult. In New Jersey, FY 1990 budget shortfalls hit COEP harder 
than it did many programs-partly because of intensive lobbying by opposing 
institutions, but partly because it was difficult to mobilize legislative support in 
hard times for a program that would only "gather information." Ironically, 
these examples provide further support for the contention that assessment and 
resource expansion are politically linked; if the legislative agenda shifts to cost 
control, most legislators feel that there are much more straightforward ways to 
identify needed cuts than through expensive and dubious testing processes. 

Creating a Vision fnr t he Future. Re-establishing credibility and making the 
case for additional resources are largely conditioned by a third element of the 
external agenda: publicly communicating a clear direction for future 
investment. Here assessment has an important role to play because the process 
itself forces unusual attention to explicit goal-setting. Commenting on this 
phenomenon most clearly was former Tennessee Governor Lamar Alexander, 
now President of the University of Tennessee: "assessment itself is not a goal 
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but it forces you to set goals," he emphasized, "and if you can clearly show 
what you're doing and how you got there, having clear goals may in itself be 
sufficient to get accountability." 

Linkages between assessment and statewide goalsetting can occur in many ways. 
In New Jersey, the COEP program was seen as having an important impact on 
delineating needed skills for the future. This, in turn, was "changing thinking" 
about how the state's system for higher education should be organized. In 
Virginia, assessment and creating an agenda proceeded in parallel SCHEV*s 
"Commission on the University of the 21st Centur/-a broad-based and highly 
visible planning effort, represents in the words of its director, "an attempt to 
seize the intellectual high ground" by creating a statewide vision for the future. 
Coupled with a decentralized assessment initiative, this approach is seen as 
heading off some of the problems of tying accountability directly to comparative 
performance. Finally, in Washington, complementary initiatives were visibly 
present in a single Master Plan. Regardless of its substantive merits, the Plan 
itself represented an important rallying point for uniting institutional and 
political interests. Indeed, institutions in Washington, most of which strongly 
opposed the Plan's sophomore testing initiative, were unusually careful about 
managing their opposition because they realized that it was in their own 
interest to ensure that the newly created and politically fragile Higher 
Education Coordinating Board was successful. 

Of equal impc nance as having a vision, however, is the ability to articulate it. 
In Illinois, where assessment efforts were embodied in a careftilly-designed 
comprehensive review of undergraduate education, legislative staff felt that the 
Illinois Board of Higher Education (IBHE) had "been successful academically, 
but not politically." Program review reporting was seen as too detailed to give 
an overadl picture of system direction; one noted favorably a brochure recently 
published by the state's association of independent colleges, and felt that a 
similar "report to shareholders" was badly needed in public higher education, 
IBHE staff, in contrast, persistently resist a "USA Today mentality" in external 
reporting, maintaining that "you can't win accountability by simply slapping 
numbers on the table." This discontinuity loomed important last spring when 
increases in higher education funding were contingent upon a new tax package, 
and when in this context the Governor called upon the IBHE to publicly 
address questions of "accountability and productivity." The sentiment behind 
this request was expressed succinctly by members of the Governor's staff: "All 
we are asking is that higher education define its goals and then tell us; how 
can they expect a tax increase if they don't show us what they are doing?" 
Although higher education in Illinois ultimately did obtain the expected tax 
increase without a "master planning" effort, this was in part due to the fact that 
the IBHE enjoyed high initial credibility with the Governor and legislature; in 
other states, such a recovery might not have been possible. 

This final external requisite of accountability, in sum, encompasses both prior 
requirements. But it adds the desirability of fusing them in the form of a set 
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of consistent, simple messages about higher educadon's plans and its 
commitment to quality. /iS demonstrated in Washington and Virgim'a, planning 
itself can be an element in establishing credibih'ty. The role of assessment in 
supporting this strategy is that it can render proposed "visions of the future" 
more concrete (and therefore more understandable), and that it suggests a 
means for actually recognizing quality attainment once it has been achieved. 

Addressing these three requisites effectively above all demands that state assessment 
policy be coherently organized in terms of an overall strategy for addressing 
undergraduate improvement This strategy may be apparent and overtly linked, as in 
Washington and to some degree in Tennessee, Florida, New Jersey and Colorado; or 
it may consist of discrete initiatives that work toward common ends as in Virginia. 
But a strategy of some kind must be actively present and communicated. Success in 
these endeavors, however, depends upon two additional factors, both of which are tied 
directly to assessment policy. 

One is the presence of a coherent state-level planning database which can be used 
both to support goal-setting and to report consistently about progress made. New 
Jersey's SURE (Student Um't Record Enrollment) reporting system, for example, was 
established at least in part to enable DHE to provide regular information of this kind. 
Similarly, performance finding in Tennessee has generated a range of useful reporting 
statistics for tracking quality improvement In Florida, moreover, lack of an integrated 
reporting system that combines CLAST test scores, placement test scores, and 
enrollment information considerably hampers efforts to determine overall policy. 
CLAST scores in particular, because they have been seen only in terms of certifying 
individual students, have been largely inaccessible as a means for evaluating system 
effectiveness. 

A second condition of success is a consistent mechanism for communicating actions 
taken to improve performance. For most legislators, the primary criterion in terms of 
which to judge the "success" of a reform is visible change in behavior. Actions taken 
in response to state initiatives are likely to speak louder than actual data on 
effectiveness. As a result, highlighting these promptly and regularly can pay 
considerable dividends. Most such actions, of course, will be at the insthutional level, 
and it is important to move examples quickly into public consciousness. Indeed, 
insthutional responsiveness itself emerged as perhaps the best single "bottom line" of 
performance among legislators in case-study states. As a Washington legislator put it, 
the clearest indicator of success would be for the insthutions themselves "to show that 
they're interested, and that they're willing to change." 



12 

le 



THE INTERNAL AGENDA FOR ACCOUNTABILITY: 
STATE POLICY AS AN AGENT FOR CHANGE 



Under the "new accountability," external and internal agendas are difficult to 
disentangle. Credibility, expansion, and vision are one side of a process that must 
also involve a range of strategies intended to actualize and institutionalize needed 
change. Implementing state-level change strategies in the arena of undergraduate 
education, however, has always been a tricl^ business because it involves carefully 
balancing external stimuli and incv itives with the recognition that lasting and effective 
action must come from within (Ewell 1985). In the reahn of assessment, case study 
states manifested this tension in many ways, conditioned by the resource and 
governance contexts within which state authorities had to operate, and by a range of 
institutional cultures. 

Nevertheless, all states faced simflar dilemmas with regard to assessment policy as an 
ingredient in quality improvement One was the difficulty of avoiding a "compliance 
response"-^ne oriented more toward answering the letter rather than toward fulfilling 
the spirit of state policy. State agency respondents, for example, often expressed the 
difficulty of getting institutions to forego "cosmic game playing" within any proposed 
set of assessment guidelines or regulations. Ironically, raising the stakes or 
articulating regulations more precisely sometimes had the unintended effect of 
isolating the institution's response from faculty and line academic administrators-the 
local agents for academic reform. In Tennessee, faculty sometimes viewed 
performance funding as administrative territory because dollars were involved; 
similarly in Virginia, at institutions with large amounts of incentive funding riding on 
development of an "acceptable" assessment plan, assessment reporting was sometimes 
seen as "too important" to allow substantial faculty participation. At the same time, 
proactive creation of an academic "Vision" for the future (as in New Jersey or 
Virginia) was often seen by insiitutions as an inappropriate state role. As a state 
higher education official put in Virgim*a, "our raising epistemological questions always 
makes the institutions nervous." 

"Internal'' requisites for accountability can also be described in terms of three broad 
policy directives. The focus is again in each case on a state-level "strategy" for 
improving quality of which assessment policy is an integral part. 

1. Recreating Standards for Student Competence and Achievement . An important 
exception to the general lack of specific perceived deficiencies in higher 
education on the part of legislators is in the area of academic standards- 
particularly in the early years of undergraduate study. Here, two distinct 
concerns are intertwined. First, legislators and lay board members are 
disturbed by fragmentary but persistent evidence about low levels of basic skills 
possessed by typical college students. In Washington, the Chairman of the 
Higher Education Coordinating Board, based on his experience as a 
businessman, had some clear opinions about shortcomings in communications 
skills, and these perceptions played a dominant role in the decision to include 
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sophomore testing in the Master Plan. Second, there are concerns about 
fragmentation in the undergraduate curriculum resulting in too little 
commonality and integration. As one Tennessee legislator put it, "all students 
ought to come out of college knowing something about the world and about 
themselves." 

To address the first concern, state policy makers were inclined at least to 
propose, and sometimes to implement, the classic K-12 solution: standardized 
testing of common outcomes. Four of the nine case-study states (Florida, 
Temiessee, New Jersey, and South Dakota) at some point invested in this 
approach, while it was concretely proposed as a policy option in at least two 
more. But only in Florida did the approach chosen completely emulate the 
K-12 solution of universal "gateway" testing. In each of the other states, testing 
was intended to provide institution-level performance information, and with the 
exception of Tennessee's use of the ACT-COMP exam as an ingredient in 
performance funding, the process did not involve high stakes for the 
institutions. 

In many ways a special case, Florida's "rising junior" CLAST examination 
illustrates both the policy benefits and drawbacks of ccrrjnon testing. Its 
-origins lie in a statewide design for undergraduate education that places 
unusual reliance on the articulated role of community colleges. Within this 
context, CLAST was intended primarily as a direct quality control device: if 
AA transfer degrees were mandated to be fully transferable, an "upper- 
divisional entrance examination" was deemed necessary to ensure consistent 
levels of preparation among transfers. In this role, for the most part, state 
policy makers in Florida believe that the examination has achieved its intended 
purpose. 

But both higher education policy leaders and those directly responsible for test 
administration acknowledge that testing mechanisms such as CLAST have 
severe limitations as general policy tools. First, because it serves as a "gate," 
CLAST is limited to the assessment of minimum skills; pass rates average over 
90% and coverage is limited to essential communications, computational and 
reasoning skills. As a result, test scores do not provide good outcomes 
measures of a multi-faceted collegiate experience. Indeed, some in Florida 
have argued that really looking at CLAST performance could actively harm 
higher education's credibility with respect to quality because the skill levels 
required for successful performance are so low. This was an issue also faced 
by other states. In New Jersey, the decision was made to develop a 
sophisticated task-based "General Intellectual Skills" (GIS) examination for 
common administration to sophomores, but a major concern was that the 
relatively high levels of skill required for successful performance on such an 
examination would unfairly discriminate against community colleges. 

Secondly, test scores have only partial utility in guiding policy because their 
actual informational content is limited. In Florida, CLAST scores are not 

14 



18 



aggregated for policy purposes and only overall pr^ss-rates are reported. 
Furthermore, because CLAST is a minimum requirement for progress, many 
students (especially in the more selective four-year institutions) are advised to 
take the examination as soon as they enroll as freshmen-completely obviating 
the possibility of using scores as collegiate outcomes measures. Use of the 
ACT-COMP examination in Tennessee has been subject to similar difficulties. 
Within the context of performance funding, the relative "unambiguity" of a test 
score as a funding device is seen as highly attractive. But concerns about 
equity in institutional allocations perpetually resurfaced as the examination 
produced results that were seen as anomalous or uninterpretable. Finally, in 
Washington, a common sophomore testing proposal was dropped after a large- 
scale pilot test suggested the limited utility of obtained results in guiding 
concrete academic quality improvements. 

Most importantly, use of a common testing mechanism can be politically 
dangerous as it creates powerful winners and losers. In Florida, this was most 
visible ir debates about whether or not higher pass-rate standards for CLAST, 
scheduled for 1989, should in fact be implemented because of adverse impact 
on minorities. In South Dakota, fears about institutional comparisons led first 
to a three-year moratorium on releasing the results of common testing, and 
ultimately to the abandonment of the program. Finally, in New Jersey, a 
severe revenue shortfall provided an occasion to solidify institutional opposition 
to the GIS examination. Resulting lobbying killed its legislative fiscal support, 
though the examination was ultimately impleme ^ted using resources drawn from 
the Governor's contingency fiind. 

These examples illustrate some dilemmas involved in using an assessment 
mechanism directly to recreate standards. More notable have been some 
positive indirect effects, whether or not they involve common outcomes 
measures. In New Jersey, a COEP task force on student learning provided a 
visible public occasion for discussing expected common levels of collegiate 
functioning. Even if no test had resulted from these deliberations, the 
discussions themselves had a notable impact on faculty opinions and behavior. 
In Florida, state efforts to address common standards proceeded in parallel 
with CLAST, primarily in the form of direct curricular interventions such as the 
"Gordon Rule." At the community colleges, one result has been more uniform 
expectations about the level and kinds of work to be expected of students 
enrolled in first-year college courses. And impacts of this kind were not 
confined to states with overt testing mechanisms. In Virginia, community 
college representatives noted considerable "cross-fertilization" among campus- 
centered assessment efforts and the deliberations of a systemwide task force on 
general education. A similar secondary effect occurred among commuiiity 
colleges in Illinois. Under the auspices of the Illinois Mathematics Association, 
math teachers at all levels have convened to determine common college 
preparatory skill levels. 
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On the one hand, assessment poh'cy can directly create higher standards 
through estabh'shing an intrusive quality control mechanism based on minimum 
standards. But its greatest long-term impact appears indirect: in inducing 
continuing discussions about what levels of performance should be expected of 
college students and about how these abilities might be concretely recognized 
and induced. 

Sending a Clear Message on Priorities State assessment policy can also exert 
a powerful effect on the ways institutions construct their own agendas with 
respect to quality undergraduate education. Indeed, it was this effect that most 
state policy makers identified as the most significant result that assessment had 
to date achieved. As one Florida legislator put it, "it's the message that we're 
sending, not the scores themselves." 

One "message" addresses the relatively low priority accorded to undergraduate 
education at most institutions, and particularly its general education component. 
Given this concern, a commonly-reported effect of assessment was simply to 
refocus attention. In Virginia, for example, an academic vice-president noted 
that assessment has "legitimized the whole conversation about undergraduate 
education." In Illinois, campus representatives observed that one impact of the 
IBHE's comprehensive review of undergraduate education had been "to force 
holistic thinking about curriculum and instruction." In both states, assessing 
general education proved particularly challenging, partly because few campus- 
level structures were in place to provide an organizational focus. In Illinois, 
this became evident when program review procedures that were normally 
anchored in academic departments were applied to undergraduate education as 
a whole. 

A second "message" centers on higher education's overall responsiveness. 
Often, this is embodied in a view that colleges and universities should become 
more visibly "service oriented" v/ith respect to their "customers." Some 
legislators initially expressed this sentiment in the form of dissatisfaction with 
the amount of faculty time typically spent in the classroom. Others referred to 
"customer complaints" about such varied issues as inaccessible faculty, complex 
bureaucratic procedures or indifferent teaching. In several states, one 
manifestation of heightened concern was an increase in legislative "provisos" 
intended to address such issues. In Illinois, for example, a bill had just been 
passed barring teaching assistants with limited English skills from the classroom. 
Here the role of assessment was seen as particularly effective in promoting 
action directed toward quality improvement. In Tennessee, required student 
surveys uncovered several such problems that could be quickly addressed. One, 
at the University of Tennessee, Knoxville, was the quality of teaching assistants. 
Student satisfaction data in this case both allowed early identification of the 
problem and, when repeated, provided convincing evidence that the institution 
had taken effective action. 
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In using assessment poliq^ to "shift attention" toward quality improvement, 
however, particular care is required to ensure that unintended messages are not . 
sent. One effectiveness indicator used in performance funding in Tennessee, 
for example, is the proportion of an institution's accreditable programs that are 
in fact accredited; an inadvertent result has been to strengthen the hand of 
professional programs at the expense of traditional disciplines. In other states, 
institutions reported that assessment and other state policies sent contradictory 
messages: while assessment policies created incentives to direct resources and 
attention toward improved instruction, parallel budgetary incentives embedded 
in other state programs led in precisely the opposite direction. 

Revitalizing Curriculum and Instructional Practice, Raising "epistemological 
questions" about instructional practice on the state level and inducing visible 
action in response is probably the most challenging ingredient of the "new 
accountability." Like its external counterparts, it involves both a changing state 
role and a consequent shift in "appropriate" organizational relationships. With 
the exception of recreating standards and demanding shifts in attention, 
however, the precise nature of these changes remains largely for institutions to 
determine. As part of an overall state strategy, therefore, the most important 
intent jf assessment policy is indirect: in the words of one legislator, "to get 
the faculty themselves to really look at what they do." 

One manifestation is simply to induce such self-examination-what a key state 
agency official in Virginia aptly termed "restlessness" in the curriculum. In 
most states, this kind of questioning occurred as a natural outcome of 
assessment once the campuses began to get on with the task. In Illinois and 
Virginia, where the architecture of assessment was unusually decentralized, this 
outcome was particularly visible. In both states, institutional representatives 
repeatedly cited positive curricular "process" effects that they felt were 
considerably more important in achieving quality than directly measurable 
"products"-effects that were probably unobtainable in other ways. Ironically, 
however, "process" effects were often predicated on prior "opposition." In 
Washington, substantive faculty involvement began only when a task force was 
formed to study (and to ultimately reject) the Master Plan's sophomore testing 
provision; opposition "coalesced" key faculty into a core of knowledge and 
interest that is now being effectively harnessed as each institution attempts to 
determine its own assessment approach. Successful efforts to mobilize faculty 
in support of assessment at Kean College in New Jersey were equally founded, 
in part, upon active opposition to COEP. Similarly, in Virginia, few faculties 
appear willing to "take the easy way out" of admini. Bering a standardized test in 
response to the state's mandate; as a result, noted SCHEV staff members, 
"the/re unable to duck the responsibility of getting really involved." By 
avoiding too specific direction, state policy makers in such cases promoted 
maximally effective local initiatives to improve teaching practice. In the best 
cases, these became, "part of the living culture of the institution," and were 
visible in its overall curricular approach. Some of the most notable examples. 
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in fact, were those in which institutions used assessment to help argue for new 
curricula in a case made to the state. 

But less directive approaches also encountered substantial limits. Here, 
perhaps the most important was uneven development due to variations in the 
commitment of institutional leadership. In each state, several examples 
emerged of institutions that used an assessment initiative proactively to address 
quality issues of local importance. "As long as faculty understand that this is 
not just intended to feed the Great MoUoch somewhere," noted an institutional 
representative in Illinois, "they see that assessment couid easily become a 
vehicle for what we've wanted to do all along." But such cases were far from 
the rule, because many institutions-at least initially-elected to adopt a strategy 
of minimum compliance. Here instiiytionai leaders felt it would be impossible 
to divorce assessment from issues of autonomy and appropriate governance. 
Choosing minimum compliance, most supported their position by citing what 
they perceived to be "intrusive" state action in response. 

The result was a real dilemma for state authorities -"ho then had to decide 
whether to escalate further and risk undermining ..leir preferred strategy of 
building local ownership for instructional improvement. In Virginia, for 
example, institution^; that had proposed nothing substantive in the assessment of 
general education were urged to adopted a .standardized test "as a placeholder"; 
this reinforced institutional claims that what the state really wanted was a 
standardized test. Furthermore, in Virginia, certifying the existence of a sound 
local process meant requiring institutions to report as evidence some kind of 
results; again this sent the expected signal that the results themselves were 
important, rather than the process of critical self-examination that produced 
them. Developing appropriate state policy under these conditions often 
involved a very careful orchestration of incentives and actions. In Virginia, 
these included adequate fiscal support for institutions undertaking high-quality 
efforts, a flexible but relatively unambiguous set of policy guidelines, and a 
good deal of reassurance. 

Other states sought to avoid the dilemma of uneven development by being 
more directive from the outset. As a THEC staff member emphasized, "in an 
RFP-like assessment process [as in Colorado or Virginia] you lose control over 
what is happening." Consequently, state policy makers increased the specificity 
of performance funding guidelines in Tennessee partly to raise the "center of 
gravity of the entire system" with respect tf » assessment practice, knowing that 
this might negatively affect such leading institutions as the University of 
Tennessee-Knoxville where local processes had already taken hold. In Florida, 
moreover, where some state officials unashamedly termed their approach a 
"coercive change modsl," the result was a comparatively uniform curricular 
response. Finally, COEP's "mix" of directive and locally-controlled elements in 
New Jersey is intended to provide minimal but visible uniformity of effort. 
DHE staff believe ultimately that "faculty, when confronted with facts, will do 
something with them," but they are also mindful of expressed legislative 



common concerns about quality performance. "We can't just have the 
institutions doing their own thing here," said one, "we have no choice." 



Many apparent differences in assessment poKcy, however, are really differences in how 
assessment is conceived within a larger state strategy of quality improvement. In 
states as different as Virginia, Tennessee, and Florida, policy makers agreed that 
assessment, in itself, was an indifferent policy tool to directly induce needed changes 
in institutional behavior. Most preferred budgetary devices such as incentive funding 
or challenge grants or, less frequently, direct regulation. But, in the words of one, 
"these things fit together." Assessment's unique contribution, most concurred, was to 
continually focus distracted faculty attention on issues of teaching practice; if and 
when a shift of attention occurred, most agreed, it was probably more effective for 
state policy to stay out of the way. 
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CONCLUSION 



These ingredients for a "new accountability" collectively constitute a significant 
challenge for higher education. Meeting them effectively will require a fundamental 
shift in perspective for the leadership of both institutions and of statewide higher 
education governing and coordinating agencies. If the "new accountability" is real, a 
substantially more proactiA^e state leadership strategy is called for-one that attempts to 
actively shape institutional agendas to embrace improvements in quality while at the 
same time providing institutions with sufficient resources and decision latitude to 
flexibly undertake needed innovations. At the same time, institutional leadership, and 
through it the faculty, must come to recognize the legitimate basis for this strategy- 
that colleges and universities must ultimately serve and reflect wider public purposes. 
At every level, considerable statesmanship will be required. 

Assessment policy constitutes the key new ingredient in a state strategy designed to 
meet the challenge of the "new accountability." Above all, this is because improving 
quality means paying attention to results. Information about results is not only a 
necessary condition for charting progress, but it also visibly signals the need for 
renewed commitment. By its very nature, assessment makes inevitable an eventual 
look at curriculum, at teaching strategies, and at academic policies-the essence of 
what colleges and universities actually do in the realm of undergraduate education. 
Other available state policy levers, while powerful, do not lead so quickly to the heart 
of the academic enterprise. At the same time, if through their own leadership 
institutions can be induced to take ownership of the process, resulting changes can be 
significant. 1 acuity too are often discontented about instructional quality, and many 
of their concerns are congruent with those of the wider public. If assessment can 
engage them on their own terms, as an educational rather than a political enterprise, 
experience has shown that enduring and appropriate changes can be expected from 
within. 

But assessment cannot do the job alone. Rather, it must work within a carefully 
crafted approach to systemic improvement. Other directed change mechanisms such 
as categorical funding or program review and approval, and traditional regulatory 
functions such as personnel policies or accounting/budgeting procedures will also have 
a part to play, and changes intended to foster quality should be expected in all of 
thero. But the key is coordination. Above all, state leadership must avoid 
constructing situations where institutions are induced to respond to such initiatives 
piecemeal 

With regard to assessment, three things in particular are required of state leadership. 
First, it must be apparent from both established policy and ongoing action that 
assessment is a commitment for the long term. Institutions cannot be expected to 
take ownership of a process that will require several years to show meaningful results, 
if in another two years there is yet another new initiative to respond to. Both faculty 
and local administrators must be able to see a long-term character in assessment 
policy-in its timelines, its levels of support, and in the integrity of the required 
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process. More importantly, faculty and institutional leadership must be shown that 
engaging m meaningful assessment pays off. Institutions that are taking the task 
seriously should see a consistency in message regarding instructional quality and 
should be promptly reinforced when what they have learned through assessment 
appears in other state communications or processes. In the long term, moreover 
consistency demands that meaningful levels of resources be directed toward exploiting 
the opportunities and addressing the problems that assessment uncovers. State leaders 
cannot expect institutions to determine and sustain quality improvement locally if they 
do not first set a consistent example. 

Finally, state leadership must sustain assessment as a process with academic integrity. 
Despite their unease about current instructional quality, faculty and institutional 
leaders have little interest in participating in an irksome exercise that they see largely 
directed toward bureaucratic ends. Assessment is different from most state policies 
because it engages institutions with what they actually do. On the one hand, this 
renders assessment unusually threatening: it places the public interest within the 
academic enterprise in a fashion that faculty find impossible to ignore. But 
assessment can also indicate genuine interest and concern about improvement if it is 
raised as a tangible part of compelling intellectual issues centered on how students 
actually learn, how curricula and teaching methods actually function, and how this 
process might collectively be improved. This will require an openness to new 
methods and new answers in assessment technology on the part of state policy. It will 
also require trust that once faculty and institutions have been given responsibility for 
improvement that they will appropriately follow through. Absent these conditions 
mandated assessment -^vill remain intellectually isolated, and improvement will become 
mere compliance. 

Properly handled, assessment policy can provide a ground upon which academic and 
public interests can legitimately converge. And managing such a convergence is the 
ultimate challenge of the "new accountability," 
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